IS THE IMAGERY DEBATE OVER? IF SO, WHAT WAS IT ABOUT?
Zenon Pylyshyn
Source: http://ruccs.rutgers.edu/faculty/pylyshyn-mehler.htm
So we seem to be faced with the proposal, which is apparently supported by neurophysiological data, that when we entertain an image we construct a literal picture in our primary visual cortex which, in turn, is manipulated by our cognitive system and examined by our visual system. Given how widespread this view has become one ought to ask whether it makes sense on internal grounds and how well it fits the large body of data that has been accumulated over the past 30 years. What is the problem with this literal picture-view?
First of all, if images correspond directly to (or are isomorphic to) topographically-organized pictorial patterns of activity in the visual cortex, this pattern would have to be three-dimensional to account for the imagery data. After all, the content and function (as well as the phenomenology) of images is clearly three-dimensional; for example the same mental scanning results are obtained in depth as in 2D (Pinker, 1980) and the phenomenon of “mental rotation” – one of the most popular demonstrations of visual imagery – is indifferent as to whether rotation occurs in the plane of the display or in depth (Shepard & Metzler, 1971). Should we then expect to find three-dimensional displays in the visual cortex? The retinotopic organization of the visual cortex is not three-dimensional in the way required (e.g., to explain scanning and rotation in depth). The spatial properties of the perceived world are not reflected in a volumetric topographical organization in the brain: as one penetrates deeper into the columnar structure of the cortical surface one does not find a representation of the third dimension of the scene. In fact, however, images are really multidimensional, insofar as they represent other spatially registered properties besides spatial patterns. For example, they represent color and luminance and motion. Are these also to be found displayed on the surface of the visual cortex?
Secondly, part of the argument for the view that a mental image consists of a topographical display in visual cortex is that the same kind of 2D cortical pattern plays a role in vision, so the visual system can play the dual role of examining the display in vision as well as in imagery. But it is more than a little dubious that visual processing involves examining such a 2D display of information about the visual world. It may well be that the visual cortex is organized retinotopically, but nothing follows from this about the form of the functional mental representations involved in vision. After all, we already knew that the retina started with a 2D display of activity, but nobody assumed that we could infer the nature of our cognitive representation of perceptual inputs from this fact. The inference from the physical structure of activity in the brain to the form of its functional representations is no more justified than would the parallel inference from a computer’s physical structural to the form of its datastructures. From a functional perspective, the argument for the involvement of a picture-like structure in visual processing is at least as problematic as the argument that such a structure is involved in mental imagery. Moreover, the fact that our phenomenal percepts appear to be laid out in a phenomenal space is irrelevant because we do not see our internal representation, we see the world as represented and it is the world we see that appears to us to be laid out in space, and for a very good reason – because it is! We can easily be misled into believing that we are examining an internal display in vision just as we are in mental imagery, but both are illusions. The evidence is quite clear that the assumption that an inner-display is constructed in vision is simply untenable (O'Regan, 1992; Pylyshyn, in preparation). Years of research on trans-saccadic integration have shown that our percepts are not built up by superimposing the information from individual glances onto a global image; indeed very little information is even retained from glance to glance and what is retained appears to be much more abstract and schematic than any picture (Irwin, 1996).
Thirdly, the idea that either vision or mental imagery involves examining a topographic display also fails to account for the fact that examining and manipulating mental images is qualitatively different from manipulating pictures in many significant ways. For example, it is the conceptual rather than graphic complexity of images that matters to how difficult an image superposition task is(see Palmer, 1977) and also to how quickly objects appear to be mentally “rotated”, see Pylyshyn, 1979).[6] Although we appear to be able to reach for imagined objects there are significant differences between our motor interaction with mental images and our motor interaction with what we see (Goodale, Jacobson, & Keillor, 1994). Also accessing information from a mental image is very different from accessing information from a scene, as many people have pointed out. To take just one simple example, we can move our gaze as well as make covert attention movements relatively freely about a scene, but not on a mental image. Try writing down a 3 x 3 matrix of letters and read them in various orders. Now imagine the matrix and try doing the same with it. Unlike the 2D matrix, some orders (e.g., the diagonal from the bottom left to the top right cell) are extremely difficult to scan on the image. If one scans one’s image the way it is alleged one does in the map-scanning experiment (Kosslyn, Ball, & Reiser, 1978), there is no reason why one should not be able to scan the matrix freely. Moreover, images do not have the signature properties of early vision; if we create images from geometrical descriptions we do not find such phenomena as spontaneous interpretation of certain 2D shapes as representing 3D objects, spontaneous reversals of bistable figures, amodal completion or subjective contours, visual illusions, as well as the incremental construction of visual interpretations and reinterpretations over time, as different aspects are noticed, and so on.[7]
I would turn this discussion of the parallels between vision and imagery around and suggest that the fact that in some situations the parallel between processing mental images and processing diagrams is so close it renders this entire line of evidence suspect, given that a real diagram and the way it is viewed using one’s eyes has properties that no mental entity and process could have. Some of the psychophysical evidence that is cited in support of a parallel between vision and mental imagery entails a similarity that is so close that it appears to attribute to the “mind’s eye” many of the properties of our own eyes. For example, it seems that the mind’s eye has a visual angle like that if a real eye (Kosslyn, 1978) and that it has a field of resolution which is also the same as our eyes; it drops off with eccentricity according to the same function and inscribes the same elliptical resolution acuity profile as that of our (real) eyes (Finke & Kosslyn, 1980; Finke & Kurtzman, 1981), and it exhibits the “oblique effect” wherein the discriminability of closely-spaced horizontal and vertical lines is superior to that of oblique lines (Kosslyn et al., 1999). Since in the case of the eye, such properties are due primarily to the structure of our retinas; these findings would suggest that the mind’s eye is similarly structured! Does the mind’s eye then have a blind spot as well? Of course, these close parallels could be just a coincidence, or it could be that the distribution of neurons and connections in the visual cortex comes to reflect the type of information it receives from the eye. But it is also possible that such phenomena reflect what people have implicitly come to know how things generally look to them, a knowledge which the experiments invite them to use in simulating what would happen in a visual situation that parallels the imagined one. Such a possibility is made al the more plausible in view of the fact that the instructions in these imagery experiments explicitly ask observers to “imagine” that they are looking at a certain situation and to imagine what it would look like to see things, say, in their peripheral vision. The fact that subjects often profess ignorance of what would happen does not establish that they do not have tacit knowledge or simply memory of similar cases that they have encountered before (see note 2).
The picture that we are being presented, of a mind’s eye gazing upon a display projected onto the visual cortex, is one that should arouse our suspicion. It comes uncomfortably close to the idea that properties of the external world, as well as of the process of vision (including the resolution pattern of the retina and the necessity of moving one’s eyes around the display to foveate features of interest), are built into the imagery system. If such properties were built in, our imagery would not be as plastic and cognitively penetrable as it is. We can after all imagine almost any properties and dynamics we like, whether or not they are physically possible, so long as we know what the situation we are imagining would look like (we can’t imagine a 4 dimensional world because we lack precisely this type of knowledge about it – we don’t know where the contours, occlusions, shadows etc would fall). The picture-theory also does not even hint at a possible neural or information-processing basis for most of the interesting phenomena of mental imagery uncovered over the past several decades, such as the efficacy of visual mnemonics, the phenomena of mental rotation, and the apparent close parallels between how things work in the world and how we imagine them to work – which makes it possible for us to plan by visualizing a process and its outcomes. The properties exhibited by our imagery do not arise by magic: if we have false beliefs about how things work, our images will exhibit false dynamics. This is exactly what happens when we imagine light of different colors being mixed, or when we imagine an object in free fall. Because most people tacitly believe in the Aristotelian mechanics of constant-velocity free fall, our imagining of free fall is inaccurate and can be shown to follow the constant-velocity trajectory (for more such examples see Pylyshyn, 1981).
Where do we stand now?
Where, then, does the “imagery debate” stand at present? As I suggested at the beginning of this essay, it all depends on what you think the debate is about. If it is supposed to be about whether reasoning using mental imagery is somehow different from reasoning without it, who can doubt that the answer must be “yes”? If it is about whether in some sense imagery involves the visual system, the answer there too must be affirmative, since imagery involves experiences similar to those produced by (and, as far as we know, only by) activity in some part of the visual system (though not in V1, according to Crick & Koch, 1995). The big questions are, of course; what part of the visual system is involved and in what way? Answering that will require a better psychological theory of the decomposition of the visual system itself. It is much too early and much too simplistic to claim that the way the visual system is deployed in visual imagery is by allowing it to look at a reconstructed retinotopic input of the sort that comes from the eye (or at least to some topographic remapping of this input).
Is the debate, as Kosslyn claims, about whether images are depictive rather than descriptive? That all depends on what you mean by “depictive”. Is any representation of geometrical, spatial, metrical or visual properties depictive? If that makes it depictive then any description of how something looks, what shape and size it is, and so on, is thereby depictive. Does being depictive require that the representation be organized spatially? That depends on what restrictions are placed on “being organized spatially”. Any physically instantiated representation is organized spatially – certainly both computer memories and books are. Does being depictive require that images “preserve metrical spatial information”, as has been claimed (Kosslyn et al., 1978)? Again that depends on what it means to “preserve” metrical space. If it means that the image must represent metrical spatial information, then any form of representation will have to do that to the extent that spatial magnitudes need to be encoded and to the extent that people do encode them. But any system of numerals, as well any analogue medium, can represent magnitudes in a useful way. If the claim that images preserve metrical spatial information means that an image uses spatial magnitudes to represent spatial magnitudes, then this is a form of the literal picture theory. And a literal picture requires not only a visual system, but a literal mind’s eye because the input is an uninterpreted layout of features.
Is there an intermediate position that we can adopt, somewhere between imagery being a symbolic representation and being a picture? This sort of representation has been the holy grail of many research programs, especially in artificial intelligence. In the case of mental imagery, the hope has been that one might develop a coherent proposal which says, in effect, that in mental imagery the visual system (or some early stage in the visual system) receives retintopically organized information that is nonetheless more abstract (or more conceptual) than a picture, but that still preserves a measure of spatial isomorphism. There is no principled reason why such a proposal could not work, if it could be properly fleshed out. But so far as I am aware nobody has even come close to making a concrete proposal for a type of representation (or a representational language) in which geometrical relations are encoded geometrically while other properties retain their symbolic force. Schemas, such as the mental models many people have discussed, represent special relations but do not have them. To have a geometrical relation would presumably require that the representation be laid out in some spatial medium, which gets us right back to the display view. The geometrical properties encoded in this way would then have to be cognitively impenetrable since they would be part of the fixed architecture. In any case this sort of “spatial schema” view of mental images would no longer be “depictive” in the straightforward intuitive sense. It would be more like a traditional semantic network or a schema, except that geometrical relations would be encoded in terms of spatial positions in some medium. Such a representation would have to be “read” just the way that sentences are read, except perhaps that proximity in the representation would have a geometrical interpretation (note that sentences too are typically encoded spatially, yet they do not use the space except to individuate and order the words). Moreover, such a spatial schema is unlikely to provide an account of such empirical phenomena as the ones described earlier – e.g., where smaller images take longer to see and distant places on an image take longer to scan to. But that is just as well since these are just the sorts of phenomena that are unlikely to be attributable to the nature of the image but to the knowledge that people have about the perceived world functions.
.
We Make it Easy to Succeed
Successwaves, Intl.
Brain Based Accelerated Success Audios
 |